Reduced power operation of time-of-flight camera

ABSTRACT

A time-of-flight (ToF) camera is configured to operate in a manner that reduces power consumption of the ToF camera. For a key frame, a key-frame depth image is generated based on a plurality of sets of key-frame IR images. Each set of key-frame IR images is acquired using a different modulation frequency of active IR light. For a P-frame after the key frame, a P-frame depth image is generated based on a set of P-frame IR images acquired using a single modulation frequency of active IR light.

BACKGROUND

A time-of-flight (ToF) camera may determine a depth of a subjectrelative to the ToF camera based on the known speed of light and ameasured time of flight of light between the ToF camera and the subject.For example, a light signal may be temporally modulated to illuminatethe subject. The back-reflected light signal may be acquired by a sensorarray of the ToF camera and evaluated to determine a phase differencefrom which the depth may be calculated.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

A time-of-flight (ToF) camera is configured to operate in a manner thatreduces power consumption of the ToF camera. For a key frame, akey-frame depth image is generated based on a plurality of sets ofkey-frame IR images. Each set of key-frame IR images is acquired using adifferent modulation frequency of active IR light. For a P-frame afterthe key frame, a P-frame depth image is generated based on a set ofP-frame IR images acquired using a single modulation frequency of activeIR light.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exploded, schematic view showing aspects of an exampletime-of-flight (ToF) camera.

FIG. 2 is a timing graph showing an example operating sequence of a ToFilluminator for acquiring a set of key-frame infrared (IR) images foreach of a plurality of different modulation frequencies for a key frame.

FIG. 3 is a timing graph showing an example operating sequence of a ToFilluminator for acquiring a set of P-frame IR images for a singlemodulation frequency for a P-frame.

FIG. 4 schematically shows an example approach for generating akey-frame depth image and subsequently generating a P-frame depth imagebased on a positional translation of features identified in thekey-frame and the P-frame.

FIG. 5 shows example ToF camera operations that are performed for aplurality P-frames between key frames.

FIG. 6 schematically shows an example approach for generating akey-frame depth image and subsequently generating a P-frame depth imagebased on a changed perspective that is determined based on motion datafrom one or more motion sensors.

FIGS. 7A-7B are a flowchart of an example method for acquiring aplurality of depth images in a manner that reduces power consumption ofa ToF camera.

FIG. 8 shows an example near-eye display device.

FIG. 9 shows an example computing system.

DETAILED DESCRIPTION

A time-of-flight (ToF) camera may determine the depth of a subject basedon the phase difference of a light signal that is temporally modulatedby a time-of-flight illuminator. However, if the subject is positionedat a distance that makes the phase difference greater than 2π, then thesubject may be confused with another object that is placed at a distancethat is a multiple of 2π. The number of periods added to the phasedifference that equates to the distance may be referred to as the ‘phasewrapping.’ To solve this issue, in one example, a ToF camera modulatesan active infrared (IR) light signal using several different modulationfrequencies to produce a sparse solution space from which the phasewrapping of the phase difference can be determined and a depth image maybe generated. However, this multi-frequency modulation approach requiresmultiple IR image captures at each of the modulation frequencies inorder to generate a depth image. Electronic components of the ToF camera(e.g., an IR laser, sensor array, microcontroller) are activated andconsume electrical power for each IR image capture. Accordingly, thegreater the number of modulation frequencies, and the greater the numberof IR image captures at each frequency, the greater the powerconsumption of the ToF camera.

Accordingly, this disclosure is directed to an approach for operating aToF camera in manner that reduces power consumption of the ToF camera.In particular, this approach includes generating key frame depth imagesfor key frames and generating one or more P-frame depth images for oneor more successive P-frames between key frames. Each key frame depthimage is generated based on a set of IR images for each of a pluralityof different modulation frequencies. Each P-frame depth image isgenerated based on a set of IR images for a single modulation frequency.The P-frame depth image may be generated using a lower number of IRimages relative to the number of IR images required to generate akey-frame depth image due to using a single modulation frequency. Assuch, the P-frame depth image may be generated using less electricalpower than the key frame depth image. By generating one or moresuccessive P-frame depth images in between key-frame depth images, powerconsumption of the ToF camera may be reduced relative to a ToF camerathat only generates depth images using the multi-frequency approach(i.e., only generating key frame depth images without generating P-framedepth images).

FIG. 1 shows aspects of an example ToF camera 100 configured to operatein the manner described above. The term ‘camera’ refers herein to anyimaging component having at least one optical aperture and sensor arrayconfigured to image a scene or subject 102. Camera 100 includes a sensorarray 104 of individually addressable sensors 106. In someimplementations, the sensors may be complementary metal-oxidesemiconductor (CMOS) elements, but other suitable architectures are alsoenvisaged. Each sensor may be responsive to light over a broadwavelength band, although this is not required. For silicon-basedsensors, the wavelength response may range from 300 to 1100 nm, forexample. Sensor array 104 is schematically illustrated with onlytwenty-five sensors 106 for simplicity, although there is no theoreticallimitation to the number of sensors 106.

Microlens array 108 optionally may be arranged directly over sensorarray 104. Microlens array 108 includes a plurality of microlenselements 110. Each microlens element 110 of microlens array 108 may beregistered to a sensor 106 of the sensor array 104. When included,microlens array 108 may provide a larger effective fill factor at eachof the sensors, for increased collection efficiency and reducedcross-talk between pixels.

A ToF illuminator 112 is configured to emit active IR light toilluminate the subject 102. In one example, the ToF illuminator 112includes an IR laser configured to emit IR light. In someimplementations, the ToF illuminator 112 optionally may include adiffuser 114 covering a field of illumination of the ToF illuminator112. In this disclosure, the term ‘infrared’ (IR) light includes socalled near-infrared (NIR) wavelengths of about 850 nm. Depthmeasurements may be taken using IR light, including NIR light, or anyother suitable wavelength. Although not shown in FIG. 1, the cameraoptionally may include a bandpass filter to limit the portion of theelectromagnetic spectrum reaching the sensors 106 to the portion of theelectromagnetic spectrum emitted by the ToF illuminator 112.

Electronic controller 116 may include a logic machine and associatedstorage machine. The storage machine may hold instructions that causethe logic machine to enact any operation, algorithm, computation, ortransformation disclosed herein. In some implementations, the logicmachine may take the form of an application-specific integrated circuit(ASIC) or system-on-a-chip (SoC), in which some or all of theinstructions are hardware- or firmware-encoded. Electronic controller116 includes a ToF controller machine 118 and an output machine 120 thatmay be operatively connected to the sensor array 104 and/or the ToFilluminator 112. Machines 118 and 120 may be implemented as separatephysical hardware and/or firmware components or incorporated into asingle hardware and/or firmware component.

The ToF controller machine 118 is configured to repeatedly activate theToF illuminator 112 and synchronously address the sensors 106 of sensorarray 104 to acquire IR images. The active light signal emitted from theToF illuminator 116 may be temporally modulated in different modulationfrequencies for different IR image captures. In the illustrated example,the ToF controller machine 118 activates the ToF illuminator 112 toilluminate the subject 102 with active IR light 122 and addresses thesensors 106 of sensor array 104 in synchronicity. IR light 122′ reflectsfrom the subject 102 back to the camera 100. The reflected IR light 122′passes through receiving optics 124 and is incident on the sensors 106of the sensor array 104 to provide a measurement. For example, themeasurement may be an intensity measurement of active IR lightback-reflected from the subject to the sensor. In the illustratedexample, IR light 122′ is measured by a sensor 106 of sensor array 104,thus providing phase information useable with the knowledge of thecamera's configuration to determine the world space position of a locusof subject 102.

The ToF controller machine 118 is configured to generate a depth image128 based on a plurality of captured IR images. The term ‘depth image’refers to an array of pixels registered to corresponding regions (X_(i),Y_(i)) of an imaged scene, with a depth value Z_(i) indicating, for eachpixel, the depth of the corresponding region. ‘Depth’ is defined as acoordinate parallel to the optical axis of the camera, which increaseswith increasing distance from the camera. The term ‘depth video’ refersherein to a time-resolved sequence of depth images. The output machine120 is configured to output the depth image 128 generated by the ToFcontroller machine 118. The output machine 120 may be configured tooutput the depth image 128 in any suitable form. In some examples, theoutput machine 120 may output the depth image 128 as a data structure inwhich each element of the matrix corresponds to a different pixel.

The ToF camera 100 is configured to generate key-frame depth imagesbased on sets of key-frame IR images acquired for the key frame. Thekey-frame sets of IR images are acquired for a plurality of differentmodulation frequencies of IR light emitted from the ToF illuminator 112.Note that a set of key-frame IR images may include one or more key-frameIR images.

FIG. 2 shows an example operating sequence of the ToF illuminator toacquire a plurality of sets of key-frame IR images for a key frame 200.In particular, during a first collection period 202 of the key frame200, the ToF illuminator is activated to emit IR light modulated at afirst modulation frequency (F1 indicated in dotted lines) to acquire afirst set of three IR images. Next, during a second collection period204 of the key frame 200, the ToF illuminator is activated to emit IRlight modulated at a second modulation frequency (F2 indicated bydot-dashed lines) to acquire a second set of three IR images. Finally,during a third collection period 206 of the key frame 200, the ToFilluminator is activated to emit IR light modulated at a thirdmodulation frequency (F3 indicated by solid lines) to acquire a thirdset of three IR images. The total collection duration of the key framemay be equally divided between the three collection periods 202, 204,206. In one example, the frame rate of the ToF camera is thirty-threeframes per second, and each of the three collection periods of the keyframe is eleven milliseconds. In the illustrated example, nine totalkey-frame IR images are acquired during collection duration of the keyframe 200. In other examples, a different number of key-frame IR imagesmay be acquired for each key frame. The three modulation frequencies(F1, F2, F3) may be any suitable different modulation frequencies. Inone example, the second modulation frequency is higher than the firstmodulation frequency and the third modulation frequency is higher thanthe second modulation frequency. In other examples, the ToF illuminatormay be modulated in a different number of modulation frequencies (e.g.,two modulation frequencies or four or more modulation frequencies).

This multi-frequency modulation approach used for the key framesrequires a relatively high number of IR image captures that causes theToF camera 100 to consume a relatively high amount of electrical power.To reduce the power consumption of the ToF camera 100, the ToF camera100 is configured to generate one or more P-frame depth images for oneor more successive P-frames between key frames. Each P-frame depth imagemay be generated based on a set of P-frame IR images acquired for theP-frame. Note that a set of P-frame IR images may include one or moreP-frame IR images. The set of P-frame IR images is acquired for a singlemodulation frequency. In one example, one third of the number of IRimages may be captured for a P-frame relative to the number of IR imagesthat are captured for a key frame.

FIG. 3 shows an example operating sequence of the ToF illuminator toacquire a set of P-frame IR images for a P-frame 300. In particular,during a first period 302 of the P-frame 300, the ToF illuminatorremains deactivated. In one example, the first period 302 of the P-frame300 may be temporally equivalent to a sum of the first collection period202 and the second collection period 204 of the key frame 200. During asecond period 304 of the P-frame 300, the ToF illuminator is activatedto emit IR light modulated at a single modulation frequency to acquire aset of three IR images. In the illustrated example, three total P-frameIR images are acquired during the total duration of the P-frame. Inother examples, a different number of P-frame IR images may be acquiredfor each P-frame.

The single modulation frequency of IR light imaged for the P-frame maybe any suitable modulation frequency. In one example, the singlemodulation frequency is equivalent to the third modulation frequency(F3) of the key frame. Typically, the modulation frequency used for theP-frame may be a high frequency. By using a high modulation frequencyfor the P-frame, a more accurate depth measurement may be obtained.Although, in some implementations, different modulation frequencies maybe used for different P-frames.

Because the ToF illuminator, the sensor array, and other electroniccomponents of the ToF camera may remain deactivated during the firstperiod 302 of the P-frame, power consumption of the ToF camera may bereduced. This may be particularly beneficial in implementations wherethe ToF camera is incorporated into a battery-powered device, such as asmartphone, a head-mounted, near-eye display device, or other mobilecomputing devices. Furthermore, because IR images are not generatedduring the first period 302 of the P-frame, processing resources of theToF camera may be directed to other processing operations that mayresult in more efficient operation of the ToF camera.

FIG. 4 schematically shows an example approach for generating akey-frame depth image and subsequently generating a P-frame depth image.In particular, for a key frame (KF¹), the ToF controller machine 118 isconfigured to acquire a plurality of sets of key-frame IR images 400 fora plurality of different modulation frequencies. In this example, thenumber of different modulation frequencies is represented by (K), andeach set of key-frame IR images includes one key-frame IR image. Akey-frame IR image for a first modulation frequency (KV¹ ₁) is depictedwith dotted lines. A key-frame IR image for a second modulationfrequency (KF¹ ₂) is depicted with dashed lines. A key-frame IR imagefor a K^(th) modulation frequency (KF¹ _(K)) is depicted in solid lines.In this example, the K^(th) modulation frequency is the highestfrequency of the plurality of different modulation frequencies.

In some implementations, various processing operations may be performedon the plurality of sets of key-frame IR images 400 to increase theaccuracy of determined depth and intensity values. In one example, theToF controller machine 118 is configured to perform an intensityde-noising operation on the plurality of sets key-frame IR images. Anysuitable intensity de-noising operation may be performed. In oneexample, the intensity de-noising operation includes applying a low-passfilter to the plurality of sets of key-frame IR images. In anotherexample, the ToF controller machine 118 is configured to perform aspatial frequency reduction operation on the plurality of sets ofkey-frame IR images. Any suitable spatial frequency reduction operationmay be performed. In another example, the ToF controller machine 118 isconfigured to perform a dynamic range reduction operation on theplurality of sets of key-frame IR images. In one example, the dynamicrange reduction operation includes applying a Napierian logarithm to theplurality of sets of key-frame IR images. The logarithm may reduce thedynamic range of the active brightness in order to reduce stretchingeffects on values in very bright or dark areas of the key-frame IRimages.

The plurality of sets of key-frame IR images 400 acquired for the keyframe (KF¹) may be used to generate a key-frame depth image 402. In theillustrated example, each set only includes a single image, althougheach set may include more images. The ToF controller machine 118 isconfigured to produce a sparse solution space from the plurality of setsof key-frame IR images 400 from which the phase wrapping of the phasedifference of the IR light is determined and the key-frame depth image402 is generated. Because, the depth information can be determined fromthe plurality of sets of key-frame IR images 400, the key-frame depthimage 402 can be generated without using information from any otherframes (e.g., other key frames or P-frames). Additionally, the ToFcontroller machine 118 may be configured to generate a key-frame IRintensity image based on the plurality of key-frame IR images 400. Thekey-frame IR intensity image may include, for each sensor of the sensorarray, an IR light intensity value.

Furthermore, the ToF controller machine 118 may identify one or morefeatures of the imaged scene based on the plurality of sets of key-frameIR images 400. Any suitable number of features may be identified and/ortracked from frame to frame. In this example, the number of differentfeatures is represented by (j). In the depicted example, in thekey-frame IR image for the K^(th) modulation frequency, a first featurehas a position (m₁ ¹, n₁ ¹), a second feature has a position (m₁ ², n₁²), and a j^(th) feature has a position (m₁ ^(j), n₁ ^(j)), where mrepresents the row, and n represents the column of the IR image. Theposition of the identified features may be tracked from the key frame tothe P-frame to determine a positional translation that may be used todetermine a phase wrapping applied to P-frame IR images acquired for theP-frame to generate a P-frame depth image.

For a P-frame (PF²) after the key frame (KF¹), the ToF controllermachine 118 is configured to acquire a set of P-frame IR images 404 fora single modulation frequency (K). In this example, the set includes asingle P-frame IR image (PF_(K) ²) depicted in solid lines. In thisexample, the single modulation frequency (K) corresponds to the highestmodulation frequency (K) of the plurality of modulation frequencies ofthe key frame (KF¹).

In some implementations, various processing operations may be performedon the set of P-frame IR images 404 to increase the accuracy ofdetermined depth and intensity values. In one example, the ToFcontroller machine 118 is configured to perform an intensity de-noisingoperation on the set of P-frame IR images. Any suitable intensityde-noising operation may be performed. In one example, the intensityde-noising operation includes applying a low-pass filter to the set ofkey-frame IR images. In another example, the ToF controller machine 118is configured to perform a spatial frequency reduction operation on theset of P-frame IR images. Any suitable spatial frequency reductionoperation may be performed. In one example, the spatial frequencyreduction operation includes applying a Napierian logarithm to the setof P-frame IR images. In another example, the ToF controller machine 118is configured to perform a dynamic range reduction operation on theplurality of sets of P-frame IR images. In one example, the dynamicrange reduction operation includes applying a Napierian logarithm to theplurality of sets of key-frame IR images.

The ToF controller machine 118 is configured to identify the features ofthe imaged scene based on the set of P-frame IR images 404. In thedepicted example, the first feature has an updated position (m₂ ¹, n₂¹), the second feature has an updated position (m_hd 2 ², n₂ ²), and thej^(th) feature has an updated position (m₂ ^(j), n₂ ^(j)) for theP-frame (PF²). The ToF controller machine 118 is configured to determinea positional translation 406 of these features from the set of key-frameIR images for the modulation frequency (K) to the set of P-frame IRimages. In one example, the positional translation 406 includes ahorizontal and vertical shift that is applied to the sets of key-frameIR images for the other modulation frequencies based on the change inposition of the tracked features such that the key-frame IR images areregistered to the P-frame IR images. After the ToF controller machine118 applies the positional translation to the sets of key-frame IRimages, the ToF controller machine 118 may be configured to crop thekey-frame IR images to match the P-frame IR images. The ToF controllermachine 118 is configured to generate a P-frame depth image 408 based atleast on the set of P-frame IR images acquired for the single modulationfrequency (K) and the positional translation of the features of thescene. More particularly, the ToF controller machine 118 may beconfigured to generate the P-frame depth image 408 also based on thetranslated and cropped key-frame IR images for the other modulationfrequencies.

This approach for generating a P-frame depth image may be repeatedlyperformed any suitable number of times for successive P-frames betweengenerating key-frame depth images for key frames. FIG. 5 shows exampleoperations that are performed for a plurality P-frames between keyframes. For the first key frame (KF¹), the ToF controller machine 118generates a key-frame depth image and a key-frame IR intensity imagebased on a plurality of sets of key-frame IR images generated for eachof a plurality of different modulation frequencies (1, 2, K).Furthermore, the ToF controller machine 118 identifies features in theplurality of sets of key-frame IR images. For a first P-frame (PF²)after the first key frame (KF¹), the ToF controller machine 118generates a set of P-frame IR images for a single modulation frequency(K). The ToF controller machine 118 identifies features in the set ofP-frame IR images, and determines a positional translation of theidentified features from the set of key-frame IR images for the highestmodulation frequency (K) to the set of P-frame IR images. In otherwords, the IR images for the same modulation frequency are compared todetermine the positional translation of the features in the scene. TheToF controller machine 118 horizontally and/or vertically shifts thekey-frame IR images for the other modulation frequencies based on thedetermined positional translation of the identified features. The ToFcontroller machine 118 crops the shifted sets of key-frame IR images forthe other modulation frequencies. The ToF controller machine 118generates the P-frame depth image based on the set of P-frame IR imagesfor the modulation frequency K of the first P-frame (PF²) and theplurality of shifted and cropped key-frame IR images for the othermodulation frequencies (1, 2) of the first key frame (KF¹).

P-frame depth images for subsequent P-frames (PF³-PF^(N)) are generatedin the same manner as the first P-frame (PF²). In particular, for eachP-frame, the positions of the features identified for the P-frame arecompared to the positions of the features in the key frame to determinethe positional translation that is applied to the plurality of key-frameIR images for the other modulation frequencies. The P-frame depth imageis generated based on the set of P-frame IR images for the P-frame andthe positional translation of the identified features from the prior keyframe to the P-frame. In some implementations, positional translationmay be intermediately tracked from P-frame to P-frame.

Any suitable number of P-frame depth images may be generated forsuccessive P-frames between key frames. In some implementations, thenumber of successive P-frames between key frames may be predetermined orfixed. For example, a fixed number ranging between three and fiveP-frame depth images may be generated between key frame depth images.

In other implementations, the ToF controller machine is configured todynamically adjust the number of successive P-frames between key frames.The number of successive P-frames between key frames may be dynamicallyadjusted based on any suitable operating parameter or condition. In oneexample, the ToF controller machine 118 is configured to dynamicallyadjust the number of successive P-frames between key frames based on anamount of positional translation of one or more features identified inkey-frame IR images and P-frame IR images. For example, if theidentified feature(s) change position by less than a threshold amountbetween the key frame and the current P-frame, then the next frame isdesignated as a P-frame and another P-frame depth image is generated. Ifthe identified feature(s) change position by greater than or equal tothe threshold amount between the key frame and the current P-frame, thenthe next frame is designated as a key frame and a key-frame depth imageis generated. According to such an approach, the depth of quick movingobjects in a scene may be accurately measured while also reducing powerconsumption of the ToF camera.

In the above described approach, a P-frame depth image is generatedbased on the positional translation of feature(s) identified in IRimages of a key frame and a P-frame. Such an approach may be implementedin a standalone ToF camera. In some implementations, a ToF camera may beincorporated into a device that includes one or more motion sensorsexternal to the ToF camera, such as a mobile computing device (e.g.,smartphone, augmented-reality device, near-eye display device). Anon-limiting example of motion sensors that are configured to determinea position of the ToF camera include an inertial measurement unit (IMU)including accelerometers and/or gyroscopes. In such implementations, theinformation from the motion sensors may be leveraged by the ToF camerato determine a change in perspective of the ToF camera between a keyframe and a P-frame that may be used to generate a P-frame depth image.In this approach, features of the scene do not need to be identified andtracked from frame to frame in order to generate a P-frame depth image.As such, this approach may be less processing resource intensive thanthe feature tracking approach. Although such feature tracking also maybe performed in some implementations.

FIG. 6 schematically shows an example approach for generating akey-frame depth image and subsequently generating a P-frame depth imagebased on a changed perspective of the ToF camera from the key frame tothe P-frame that is determined based on motion sensor data. Inparticular, for a key frame (KF¹), the ToF controller machine 118 isconfigured to acquire a plurality of sets of key-frame IR images 600 foreach of a plurality of different modulation frequencies. Note that eachset of key-frame IR images may include one or more key-frame IR images.The plurality of sets of key-frame IR images 600 acquired for the keyframe (KF¹) may be used to generate a key-frame depth image 602. Inparticular, the ToF controller machine 118 is configured to produce asparse solution space from the plurality of key-frame IR images 600 fromwhich the phase wrapping of the phase difference of the IR light isdetermined and the key-frame depth image 602 is generated.

For a P-frame (PF²) after the key frame (KF¹), the ToF controllermachine 118 is configured to acquire a set of P-frame IR images 604 fora single modulation frequency (K). A set including a single P-frame IRimage (PF_(K) ²) is depicted. In this example, the single modulationfrequency (K) corresponds to the highest modulation frequency (K) of theplurality of modulation frequencies of the key frame (KF1). Thecontroller machine 118 is configured to receive motion data 606 from themotion sensor(s). The ToF controller machine 118 is configured todetermine a changed perspective of the ToF camera based on detectedmotion from the key frame (KF¹) to the P-frame (PF²). The ToF controllermachine 118 is configured to shift the key-frame IR images of the setsfor the other modulation frequencies such the key-frame IR images areregistered to the set of P-frame IR images based on the determinedchanged perspective of the ToF camera. The ToF controller machine 118 isconfigured to crop the shifted key-frame IR images. The ToF controllermachine 118 is configured to generate the P-frame depth image 608 basedon the set of P-frame IR images for the modulation frequency K of thefirst P-frame (PF²) and the sets of shifted and cropped key-frame IRimages for the other modulation frequencies (1, 2) of the first keyframe (KF¹). This approach for generating a P-frame depth image may berepeatedly performed any suitable number of times for successiveP-frames between generating key-frame depth images for key frames.

FIGS. 7A-7B depict a flowchart of an example method 700 for generatingdepth images in a manner that reduces power consumption of a ToF camera.For example, method 700 may be enacted by electronic controller 116 ofcamera 100.

In FIG. 7A, at 702 of method 700, for a key frame, a ToF illuminator ofa ToF camera is repeatedly activated to illuminate a scene with activeIR light. The ToF modulates the active IR light in a plurality ofdifferent modulation frequencies. At 704 of method 700, for each of theplurality of different modulation frequencies, a sensor array of the ToFcamera is repeatedly addressed to acquire a set of key-frame IR imagesthat represent measurements of the active IR light reflected from thescene back to the sensor array. At 706 of method 700, a key-frame depthimage is generated based on a plurality of sets of key-frame IR images.Each set of key-frame IR images is acquired using a different modulationfrequency of active IR light. The key frame depth image includes a depthvalue for each sensor of the sensor array.

In some implementations, at 708 of method 700, an intensity de-noisingoperation optionally may be performed on the plurality of sets ofkey-frame IR images. At 710 of method 700, a spatial frequency reductionoperation optionally may be performed on the plurality of sets ofkey-frame IR images. At 712 of method 700, one or more features of thescene optionally may be identified based on the plurality of sets ofkey-frame IR images. The feature(s) may be identified in implementationswhere a positional translation of the identified features is used togenerate a P-frame depth image.

At 714 of method 700, a key-frame depth image is output from the ToFcamera. In some implementations, at 716 of method 700, a key frame IRintensity image optionally may be output from the ToF camera.

In FIG. 7B, at 718 of method 700, for a P-frame, the ToF illuminator isactivated to illuminate the scene with active IR light in a singlemodulation frequency. At 720 of method 700, the sensor array isaddressed to acquire a set of P-frame IR images that representsmeasurements of the active IR light reflected from the scene back to thesensor array.

In some implementations, at 722 of method 700, an intensity de-noisingoperation optionally may be performed on the set of P-frame IR images.At 724 of method 700, a spatial frequency reduction operation optionallymay be performed on the set of P-frame IR images. At 726 of method 700,one or more features of the scene optionally may be identified based theset of P-frame IR images. At 728 of method 700, a positional translationof the one or more identified features from the key-frame to the P-frameoptionally may be determined. In implementations where features areidentified in the IR images, the positional translation of theidentified features may be applied to sets of key-frame IR images forthe other modulation frequencies, and these translated key-frame IRimages may be used to generate the P-frame depth image. At 730 of method700, a changed perspective of the ToF camera for the P-frame relative tothe key frame optionally may be determined based on motion data of oneor more motion sensors. In implementations where motion data is receivedfrom one or more motion sensors, the changed perspective of the ToFcamera determined from the motion data may be used to translate the setsof key-frame IR images for the other modulation frequencies, and thesetranslated sets of key-frame IR images may be used to generate theP-frame depth image. At 734 of method 700, the sets of key-frame IRimages for the other modulation frequencies optionally may be croppedbased on the positional translation of the identified features or thechanged perspective of the ToF camera.

At 734 of method 700, a P-frame depth image is generated based on theset of P-frame IR images and the positional translation of theidentified features or the changed perspective of the ToF camera. TheP-frame depth image includes depth values for each sensor of the sensorarray. In implementations where the identified features are tracked fromthe key frame to the P-frame, the sets of key-frame IR images that aretranslated based on the positional translation of the identifiedfeatures are used with the set of P-frame IR images to generate theP-frame depth image. In implementations where the perspective of the ToFcamera is tracked from the key frame to the P-frame, the sets ofkey-frame IR images that are translated based on the changed perspectiveof the ToF camera are used with the set of P-frame IR images to generatethe P-frame depth image.

At 736 of method 700, the P-frame depth image is output from the ToFcamera. In some implementations, at 738 of method 700, a P-frame IRintensity image optionally may be output from the ToF camera. Portionsof method 700 may be repeated to generate key-frame depth images for keyframes and P-frame depth images for one or more successive P-framesbetween key frames.

FIG. 8 shows aspects of a near-eye display device 800 in which a ToFcamera may be incorporated. Near-eye display device 800 is a binocular,near-eye display device with see-thru display windows 802R and 802L anda frame 804 configured to rest on a user's head. Near-eye display device800 includes right microdisplay 806R and left microdisplay 806L. Theright and left microdisplays are configured to project computerizedvirtual display imagery onto right and left display windows 802R and802L, respectively. The microdisplays are driven by an on-board computer808. When the right and left display images are composed and presentedin an appropriate manner on display windows 802R and 802L, the userexperiences an illusion of one or more virtual objects at specifiedpositions, and having specified 3D content and other display properties.Such virtual imagery may have any desired complexity; it may, forexample, comprise a complete virtual scene having both foreground andbackground portions.

Near-eye display device 800 includes a sensor subsystem 810 operativelycoupled to computing system 806. Sensor subsystem 808 includes aworld-facing ToF camera 812 configured to image any or all aspects ofthe user's environment, including one or more real objects. For example,depth images from the world-facing ToF camera 812 may be provided tocomputing system 808, for the purpose of reconstructing the environmentvirtually. Sensor subsystem 810 may include a discrete flat-imagingcamera 814 arranged with an optical axis oriented in the same directionas an optical axis of ToF camera 812. In some implementations, image orvideo output from the flat-imaging camera and output from the ToF cameramay be co-registered and combined into a unitary (e.g., RGB+depth) datastructure or stream. In some examples, a data stream representing bothdepth and brightness (e.g., IR+depth) may be available by combiningoutputs differing in phase.

Sensor subsystem includes a position sensor 816 configured to sense aposition and orientation of the near-eye display device 800 relative toan object in the environment, or to some other locus of reference. Theposition sensor may include an inertial measurement unit (IMU) includingone or more accelerometers, gyroscopes, and magnetometers, and/or aglobal positioning system (GPS) receiver. In some implementations, theposition sensor returns a six degrees-of-freedom (6DOF) estimate of thethree Cartesian coordinates of the near-eye display device, plus arotation about each of the three Cartesian axes. The output of theposition sensor may be used to map the position, size, and orientationof virtual display objects (defined globally) onto the right and leftdisplay windows 802 of the near-eye display device 800. Sensor subsystem810 may include any suitable type of sensor including one or more motionsensors configured to determine a position and/or orientation of thenear-eye display device 800.

Sensors of sensor subsystem 810 may be configured to send sensor data tothe ToF camera 812 to indicate a changed perspective of the ToF camera812 that may be used by the ToF camera 812 to generate P-frame depthimages in the manner described herein.

In some implementations, the methods and processes described herein maybe tied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 9 schematically shows a non-limiting implementation of a computingsystem 900 that can enact one or more of the methods and processesdescribed above. Computing system 900 is shown in simplified form. Forexample, computing system 900 may take the form of camera 100 orelectronic controller 116 of FIG. 1.

Computing system 900 includes a logic machine 902 and a storage machine904. Computing system 900 may optionally include a display subsystem906, input subsystem 908, communication subsystem 910, and/or othercomponents not shown in FIG. 900.

Logic machine 902 includes one or more physical devices configured toexecute instructions. For example, the logic machine 902 may beconfigured to execute instructions that are part of one or moreapplications, services, programs, routines, libraries, objects,components, data structures, or other logical constructs. Suchinstructions may be implemented to perform a task, implement a datatype, transform the state of one or more components, achieve a technicaleffect, or otherwise arrive at a desired result.

The logic machine 902 may include one or more processors configured toexecute software instructions. Additionally or alternatively, the logicmachine 902 may include one or more hardware or firmware logic machinesconfigured to execute hardware or firmware instructions. Processors ofthe logic machine 902 may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic machine optionally may be distributed among two or more separatedevices, which may be remotely located and/or configured for coordinatedprocessing. Aspects of the logic machine 902 may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration.

Storage machine 904 includes one or more physical devices configured tohold instructions executable by the logic machine 902 to implement themethods and processes described herein. When such methods and processesare implemented, the state of storage machine 904 may betransformed—e.g., to hold different data.

Storage machine 904 may include semiconductor memory (e.g., RAM, EPROM,EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive,floppy-disk drive, tape drive, MRAM, etc.), among others. Storagemachine 904 may include volatile, nonvolatile, dynamic, static,read/write, read-only, random-access, sequential-access,location-addressable, file-addressable, and/or content-addressabledevices.

It will be appreciated that storage machine 904 includes one or morephysical devices. However, aspects of the instructions described hereinalternatively may be propagated by a communication medium (e.g., anelectromagnetic signal, an optical signal, etc.) that is not held by aphysical device for a finite duration.

Aspects of logic machine 902 and storage machine 904 may be integratedtogether into one or more hardware-logic components. Such hardware-logiccomponents may include field-programmable gate arrays (FPGAs), program-and application-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

When included, display subsystem 906 may be used to present a visualrepresentation of data held by storage machine 904. This visualrepresentation may take the form of display images translating matrix ofpixels 126 into a visual format perceivable by a human. As the hereindescribed methods and processes change the data held by the storagemachine, and thus transform the state of the storage machine, the stateof display subsystem 906 may likewise be transformed to visuallyrepresent changes in the underlying data. Display subsystem 906 mayinclude one or more display devices utilizing virtually any type oftechnology. Such display devices may be combined with logic machine 902and/or storage machine 904 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 908 may comprise or interface with one ormore user-input devices such as a keyboard, mouse, touch screen, or gamecontroller. In some embodiments, the input subsystem may comprise orinterface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; as well as electric-field sensing componentry for assessingbrain activity.

When included, communication subsystem 910 may be configured tocommunicatively couple computing system 900 with one or more othercomputing devices. Communication subsystem 910 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem 910 may be configured for communication via a wirelesstelephone network, or a wired or wireless local- or wide-area network.In some embodiments, the communication subsystem 910 may allow computingsystem 900 to send and/or receive messages to and/or from other devicesvia a network such as the Internet.

In an example, a time-of-flight (ToF) camera comprises a ToF illuminatorconfigured to emit active IR light, a sensor array including a pluralityof sensors each configured to measure active IR light, and a ToFcontroller machine configured to, for a key frame, repeatedly activatethe ToF illuminator to illuminate a scene with active IR light, whereinthe ToF illuminator modulates the active IR light in a plurality ofdifferent modulation frequencies, for each of the plurality of differentmodulation frequencies, repeatedly address the sensor array to acquire aset of key-frame IR images that each represent measurements of theactive IR light reflected from the scene back to the sensor array,generate a key-frame depth image that includes, for each sensor of thesensor array, a depth value based on a plurality of sets of key-frame IRimages, each set of key-frame IR images acquired using a differentmodulation frequency of active IR light, and identify one or morefeatures of the scene based on the plurality of sets of key-frame IRimages, for a P-frame occurring after the key frame, activate the ToFilluminator to illuminate the scene with active IR light modulated in asingle modulation frequency, address the sensor array to acquire a setof P-frame IR images that represents measurement of the active IR lightreflected from the scene back to the sensor array, identify the one ormore features of the scene based on the set of P-frame IR images,determine a positional translation of the one or more features from thekey-frame to the P-frame, and generate a P-frame depth image thatincludes, for each sensor of the sensor array, a depth value based atleast on the set of P-frame IR images acquired using the singlemodulation frequency of active IR light and the positional translationof the one or more features of the scene, and an output machineoperatively connected to the sensor array and configured to output thekey-frame depth image and the P-frame depth image. In this exampleand/or other examples, the ToF controller machine may be configured togenerate a plurality of P-frame depth images for successive P-framesbetween key frames. In this example and/or other examples, the ToFcontroller machine may be configured to dynamically adjust a number ofsuccessive P-frames between key frames based on an amount of positionaltranslation of the one or more features. In this example and/or otherexamples, the single modulation frequency may be a highest modulationfrequency of the plurality of different modulation frequencies. In thisexample and/or other examples, the ToF controller machine may beconfigured to generate the P-frame depth image also based on the setskey-frame IR images for the modulation frequencies other than the singlemodulation frequency of the P-frame. In this example and/or otherexamples, the ToF controller machine may be configured to crop thekey-frame IR images for the modulation frequencies other than the singlemodulation frequency based on the positional translation of the one ormore features, and the ToF controller machine may be configured togenerate the P-frame depth image also based on the cropped key-frame IRimages. In this example and/or other examples, the ToF controllermachine may be configured to perform an intensity de-noising operationon the plurality of sets of key-frame IR images and the set of P-frameIR images. In this example and/or other examples, the intensityde-noising operation may include applying a low-pass filter to theplurality of sets of key-frame IR images and the sets of P-frame IRimages. In this example and/or other examples, the ToF controllermachine may be configured to perform a spatial frequency reductionoperation on the plurality of sets of key-frame IR images and the set ofP-frame IR images. In this example and/or other examples, the ToFcontroller machine may be configured to perform a dynamic rangereduction operation on the plurality of sets of key-frame IR images andthe set of P-frame IR images.

In an example, a ToF camera comprises a ToF illuminator configured toemit active IR light, a sensor array including a plurality of sensors,and a ToF controller machine configured to for a key frame, generate akey-frame depth image that includes, for each sensor of the plurality ofsensors of the sensor array, a depth value based on a plurality of setsof key-frame IR images, each set of key-frame IR images acquired using adifferent modulation frequency of active IR light, for a P-frameoccurring after the key frame, generate a P-frame depth image thatincludes, for each sensor of the plurality of sensors of the sensorarray, a depth value based on a single set of P-frame IR images acquiredusing a single modulation frequency of active IR light, and an outputmachine operatively connected to the sensor array and configured tooutput the key-frame depth image and the P-frame depth image. In thisexample and/or other examples, the ToF controller machine may beconfigured to, for the key frame, repeatedly activate the ToFilluminator to illuminate a scene with active IR light, wherein the ToFilluminator modulates the active IR light in a plurality of differentmodulation frequencies, for each of the plurality of differentmodulation frequencies, repeatedly address the sensor array to acquire aset of key-frame IR images that represent measurements of the active IRlight reflected from the scene back to the sensor array, for each sensorof the plurality of sensors of the sensor array, determine a depth valuebased on the plurality of sets of IR images, identify one or morefeatures of the scene based on the plurality of sets of key-frame IRimages, for the P-frame, activate the ToF illuminator to illuminate thescene with active IR light modulated in the single modulation frequency,address the sensor array to acquire the set of P-frame IR images thatrepresents measurement of the active IR light reflected from the sceneback to the sensor array, identify the one or more features of the scenebased on the set of P-frame IR images, determine a positionaltranslation of the one or more features from the key-frame to theP-frame, and wherein each depth value of the P-frame depth image isdetermined based on the single set of P-frame IR images and thepositional translation of the one or more features of the scene. In thisexample and/or other examples, the ToF camera may further comprise oneor more motion sensors configured to measure a position of the ToFcamera, and the ToF controller machine may be configured to for the keyframe, repeatedly activate the ToF illuminator to illuminate a scenewith active IR light, wherein the ToF illuminator modulates the activeIR light in a plurality of different modulation frequencies, for each ofthe plurality of different modulation frequencies, repeatedly addressthe sensor array to acquire a set of key-frame IR images that representmeasurements of the active IR light reflected from the scene back to thesensor array, for each sensor of the plurality of sensors of the sensorarray, determine a depth value based on the plurality of sets of IRimages, for the P-frame, activate the ToF illuminator to illuminate thescene with active IR light modulated in the single modulation frequency,address the sensor array to acquire a set of IR images that representmeasurements of the active IR light reflected from the scene back to thesensor array, determine a changed perspective of the ToF camera from thekey frame to the P-frame based on motion data of the one or more motionsensors, and wherein each depth value of the P-frame depth image isdetermined based on the single set of P-frame IR images and the changedperspective of the ToF camera. In this example and/or other examples,the ToF controller machine may be configured to generate a plurality ofP-frame depth images for successive P-frames between key frames. In thisexample and/or other examples, the ToF controller machine may beconfigured to dynamically adjust a number of successive P-frames betweenkey frames based on an amount that a perspective of the ToF camerachanges from the key frame to the P-frame. In this example and/or otherexamples, the ToF controller machine may be configured to dynamicallyadjust a number of successive P-frames between key frames based on anamount that one or more features identified in the plurality of sets ofkey frame IR images and the set of P-frame IR images moves from the keyframe to the P-frame. In this example and/or other examples, the ToFcontroller machine may be configured to generate the P-frame depth imagealso based on the sets of key-frame IR images for the modulationfrequencies other than the single modulation frequency of the P-frame.In this example and/or other examples, the ToF controller machine may beconfigured to crop the key-frame IR images for the modulationfrequencies other than the single modulation frequency based on thepositional translation of the changed perspective of the ToF camera, andthe ToF controller machine may be configured to generate the P-framedepth image also based on the cropped key-frame IR images. In thisexample and/or other examples, the ToF controller machine may beconfigured to perform one or more of an intensity de-noising operationon the plurality of key-frame IR images, perform a spatial frequencyreduction operation on the plurality of key-frame IR images and theplurality of P-frame IR images, and a dynamic range reduction operationon the plurality of sets of key-frame IR images and the set of P-frameIR images.

In an example, a time-of-flight (ToF) method, comprises generating akey-frame depth image that includes, for each of a plurality of sensorsin a sensor array, a depth value based on a plurality of sets ofkey-frame (infrared) IR images, each set of key-frame IR images acquiredusing a different modulation frequency of active IR light, andgenerating a P-frame depth image that includes, for each of theplurality of sensors in the sensor array, a depth value based on asingle set of P-frame IR images acquired using a single modulationfrequency of active IR light.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A time-of-flight (ToF) camera comprising: a ToF illuminatorconfigured to emit active IR light; a sensor array including a pluralityof sensors each configured to measure active IR light; and a ToFcontroller machine configured to: for a key frame: repeatedly activatethe ToF illuminator to illuminate a scene with active IR light, whereinthe ToF illuminator modulates the active IR light in a plurality ofdifferent modulation frequencies, for each of the plurality of differentmodulation frequencies, repeatedly address the sensor array to acquire aset of key-frame IR images that each represent measurements of theactive IR light reflected from the scene back to the sensor array,generate a key-frame depth image that includes, for each sensor of thesensor array, a depth value based on a plurality of sets of key-frame IRimages, each set of key-frame IR images acquired using a differentmodulation frequency of active IR light, and identify one or morefeatures of the scene based on the plurality of sets of key-frame IRimages, for a P-frame occurring after the key frame: activate the ToFilluminator to illuminate the scene with active IR light modulated in asingle modulation frequency, address the sensor array to acquire a setof P-frame IR images that represents measurement of the active IR lightreflected from the scene back to the sensor array, identify the one ormore features of the scene based on the set of P-frame IR images,determine a positional translation of the one or more features from thekey-frame to the P-frame, and generate a P-frame depth image thatincludes, for each sensor of the sensor array, a depth value based atleast on the set of P-frame IR images acquired using the singlemodulation frequency of active IR light and the positional translationof the one or more features of the scene; and an output machineoperatively connected to the sensor array and configured to output thekey-frame depth image and the P-frame depth image.
 2. The ToF camera ofclaim 1, wherein the ToF controller machine is configured to generate aplurality of P-frame depth images for successive P-frames between keyframes.
 3. The ToF camera of claim 1, wherein the ToF controller machineis configured to dynamically adjust a number of successive P-framesbetween key frames based on an amount of positional translation of theone or more features.
 4. The ToF camera of claim 1, wherein the singlemodulation frequency is a highest modulation frequency of the pluralityof different modulation frequencies.
 5. The ToF camera of claim 1,wherein the ToF controller machine is configured to generate the P-framedepth image also based on the sets key-frame IR images for themodulation frequencies other than the single modulation frequency of theP-frame.
 6. The ToF camera of claim 5, wherein the ToF controllermachine is configured to crop the key-frame IR images for the modulationfrequencies other than the single modulation frequency based on thepositional translation of the one or more features, and wherein the ToFcontroller machine is configured to generate the P-frame depth imagealso based on the cropped key-frame IR images.
 7. The ToF camera ofclaim 1, wherein the ToF controller machine is configured to perform anintensity de-noising operation on the plurality of sets of key-frame IRimages and the set of P-frame IR images.
 8. The ToF camera of claim 7,wherein the intensity de-noising operation includes applying a low-passfilter to the plurality of sets of key-frame IR images and the sets ofP-frame IR images.
 9. The ToF camera of claim 1, wherein the ToFcontroller machine is configured to perform a spatial frequencyreduction operation on the plurality of sets of key-frame IR images andthe set of P-frame IR images.
 10. The ToF camera of claim 9, wherein theToF controller machine is configured to perform a dynamic rangereduction operation on the plurality of sets of key-frame IR images andthe set of P-frame IR images.
 11. A ToF camera comprising: a ToFilluminator configured to emit active IR light; a sensor array includinga plurality of sensors; and a ToF controller machine configured to: fora key frame, generate a key-frame depth image that includes, for eachsensor of the plurality of sensors of the sensor array, a depth valuebased on a plurality of sets of key-frame IR images, each set ofkey-frame IR images acquired using a different modulation frequency ofactive IR light, for a P-frame occurring after the key frame, generate aP-frame depth image that includes, for each sensor of the plurality ofsensors of the sensor array, a depth value based on a single set ofP-frame IR images acquired using a single modulation frequency of activeIR light; and an output machine operatively connected to the sensorarray and configured to output the key-frame depth image and the P-framedepth image.
 12. The ToF camera of claim 11, wherein the ToF controllermachine is configured to: for the key frame: repeatedly activate the ToFilluminator to illuminate a scene with active IR light, wherein the ToFilluminator modulates the active IR light in a plurality of differentmodulation frequencies, for each of the plurality of differentmodulation frequencies, repeatedly address the sensor array to acquire aset of key-frame IR images that represent measurements of the active IRlight reflected from the scene back to the sensor array, for each sensorof the plurality of sensors of the sensor array, determine a depth valuebased on the plurality of sets of IR images, identify one or morefeatures of the scene based on the plurality of sets of key-frame IRimages, for the P-frame: activate the ToF illuminator to illuminate thescene with active IR light modulated in the single modulation frequency,address the sensor array to acquire the set of P-frame IR images thatrepresents measurement of the active IR light reflected from the sceneback to the sensor array, identify the one or more features of the scenebased on the set of P-frame IR images, determine a positionaltranslation of the one or more features from the key-frame to theP-frame, and wherein each depth value of the P-frame depth image isdetermined based on the single set of P-frame IR images and thepositional translation of the one or more features of the scene.
 13. TheToF camera of claim 11, further comprising: one or more motion sensorsconfigured to measure a position of the ToF camera; wherein the ToFcontroller machine is configured to: for the key frame: repeatedlyactivate the ToF illuminator to illuminate a scene with active IR light,wherein the ToF illuminator modulates the active IR light in a pluralityof different modulation frequencies, for each of the plurality ofdifferent modulation frequencies, repeatedly address the sensor array toacquire a set of key-frame IR images that represent measurements of theactive IR light reflected from the scene back to the sensor array, foreach sensor of the plurality of sensors of the sensor array, determine adepth value based on the plurality of sets of IR images, for theP-frame: activate the ToF illuminator to illuminate the scene withactive IR light modulated in the single modulation frequency, addressthe sensor array to acquire a set of IR images that representmeasurements of the active IR light reflected from the scene back to thesensor array, determine a changed perspective of the ToF camera from thekey frame to the P-frame based on motion data of the one or more motionsensors, and wherein each depth value of the P-frame depth image isdetermined based on the single set of P-frame IR images and the changedperspective of the ToF camera.
 14. The ToF camera of claim 11, whereinthe ToF controller machine is configured to generate a plurality ofP-frame depth images for successive P-frames between key frames.
 15. TheToF camera of claim 14, wherein the ToF controller machine is configuredto dynamically adjust a number of successive P-frames between key framesbased on an amount that a perspective of the ToF camera changes from thekey frame to the P-frame.
 16. The ToF camera of claim 14, wherein theToF controller machine is configured to dynamically adjust a number ofsuccessive P-frames between key frames based on an amount that one ormore features identified in the plurality of sets of key frame IR imagesand the set of P-frame IR images moves from the key frame to theP-frame.
 17. The ToF camera of claim 11, wherein the ToF controllermachine is configured to generate the P-frame depth image also based onthe sets of key-frame IR images for the modulation frequencies otherthan the single modulation frequency of the P-frame.
 18. The ToF cameraof claim 17, wherein the ToF controller machine is configured to cropthe key-frame IR images for the modulation frequencies other than thesingle modulation frequency based on the positional translation of thechanged perspective of the ToF camera, and wherein the ToF controllermachine is configured to generate the P-frame depth image also based onthe cropped key-frame IR images.
 19. The ToF camera of claim 11, whereinthe ToF controller machine is configured to perform one or more of anintensity de-noising operation on the plurality of key-frame IR images,a spatial frequency reduction operation on the plurality of key-frame IRimages and the plurality of P-frame IR images, and a dynamic rangereduction operation on the plurality of sets of key-frame IR images andthe set of P-frame IR images.
 20. A time-of-flight (ToF) method,comprising: generating a key-frame depth image that includes, for eachof a plurality of sensors in a sensor array, a depth value based on aplurality of sets of key-frame (infrared) IR images, each set ofkey-frame IR images acquired using a different modulation frequency ofactive IR light; and generating a P-frame depth image that includes, foreach of the plurality of sensors in the sensor array, a depth valuebased on a single set of P-frame IR images acquired using a singlemodulation frequency of active IR light.